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REMARKS 

This Petition is to request (i) that above referenced 
application, which was abandoned on June 10, 2005 due to the 
failure to timely respond to a Non-final Office Action dated 
April 21, 2004, and having a response time up to and including 
July 21, 2004, be revived; (ii) that the petition fee set 
forth under MPEP 711.03(c) II submitted herewith be accepted, 
and (iii) that the AMENDMENT IN RESPONSE TO THE NON- FINAL 
OFFICE ACTION submitted herewith be accepted. 

1. ) A Non- final Office Action for the present application 

was mailed by the U.S.P.T.O. (the "Office") on April 21, 
2004. 

2. ) An amendment in response to the Non-final Office Action 

was mailed by the Applicants on July 19, 2004. The 
response also included a return postcard, which was 
stamped by U.S.P.T.O as received on July 21, 2 004. 

3. ) A Notice of Abandonment was mailed by the Office on June 

10, 2004. 

4. ) Copies of the Notice of Abandonment, the Return 

Postcard, and the AMENDMENT IN RESPONSE TO THE NON-FINAL 
OFFICE ACTION are submitted herewith. 

5. ) The Assistant Commissioner for Patents is hereby 

authorized to charge Account No. 50-0749 for the 
Petition Fee under 37 CFR 1.17 (m) in the amount of 
$130.00. 

6. ) Because the application was filed after June 8, 1995, no 

terminal disclaimer or disclaimer fee is required. 
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In view of the foregoing, it is respectfully submitted 
that the response was timely filed. Accordingly, it is 
respectfully requested that (i) the holding of abandonment be 
withdrawn, and (ii) the active status of the above referenced 
application be acknowledged. 

It is respectfully requested that the undersigned be 
contacted by telephone at the below listed telephone number, 
in order to expedite resolution of any remaining issues, if 
any further comments, questions or suggestions arise in 
connection with the application. 

To the extent necessary, a petition for an extension of 
time under 37 C.F.R. 1.136 is hereby made. Please charge any 
shortage in fees due in connection with the filing of this 
paper, including petition and extension of time fees, to 
Deposit Account 50-0749 and please credit any excess fees to 
such deposit account. 



Andrew J. Curt in 
Registration No. 48,485 




C. 



201 Broadway, 8 Floor 
Telephone (617) -621-7573 
DATE: September 8, 2 004 
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Application N 
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Applicant(s) 
PEKER ET AL 



Examiner 
Wenpeng Chen 



Art Unit 

2624 



The MAILING DA TE of this communication appears on the cover sheet with the correspondence address^ 

This application is abandoned in view of: 

1 . Applicant's failure to timely file a proper reply to the Office letter mailed on 21 April 2004 . 

(a) □ A reply was received on (with a Certificate of Mailing or Transmission dated ), which is after the expiration of the 

period for reply (including a total extension of time of month(s)) which expired on . 

(b) □ A proposed reply was received on , but it does not constitute a proper reply under 37 CFR 1.1 13 (a) to the final rejection. 

(A proper reply under 37 CFR 1 . 1 1 3 to a final rejection consists only of: (1 ) a timely filed amendment which places the 
application in condition for allowance; (2) a timely filed Notice of Appeal (with appeal fee); or (3) a timely filed Request for 
Continued Examination (RCE) in compliance with 37 CFR 1.114). 

(c) □ A reply was received on but it does not constitute a proper reply, or a bona* fide attempt at a proper reply, to the non- 

final rejection. See 37 CFR 1.85(a) and 1.1 1 1. (See explanation in box 7 below). 

(d) S No reply has been received. 

2. □ Applicant's failure to timely pay the required issue fee and publication fee, if applicable, within the statutory period of three months 

from the mailing date of the Notice of Allowance (PTOL-85). 

(a) □ The issue fee and publication fee, if applicable, was received on (with a Certificate of Mailing or Transmission dated 

), which is after the expiration of the statutory period for payment of the issue fee (and publication fee) set in the Notice of 

Allowance (PTOL-85). 

(b) □ The submitted fee of $ is insufficient. A balance of $ is due. 

The issue fee required by 37 CFR 1.18 is $ . The publication fee, if required by 37 CFR 1.18(d), is $ . 

(c) □ The issue fee and publication fee, if applicable, has not been received. 

3-D Applicant's failure to timely file corrected drawings as required by, and within the three-month period set in, the Notice of 
Allowability (PTO-37). 

(a) □ Proposed corrected drawings were received on (with a Certificate of Mailing or Transmission dated ), which is * 

after the expiration of the period for reply. 

(b) □ No corrected drawings have been received. 

4. □ The letter of express abandonment which is signed by the attorney or agent of record, the assignee of the entire interest, or all of 

the applicants. 

5. □ The letter of express abandonment which is signed by an attorney or agent (acting in a representative capacity under 37 CFR 

1 .34(a)) upon the filing of a continuing application. 



6. □ The decision by the Board of Patent Appeals and Interference rendered on . 
of the decision has expired and there are no allowed claims. 

7. □ The reason(s) below: 



and because the period for seeking court review 



Wenpeng Chen 
Primary Examiner 
Art Unit: 2624 



Petitions to revive under 37 CFR 1.137(a) or (b), or requests to withdraw the holding of abandonment under 37 CFR 1.181 , should be promptly filed to 
minimize an y negative effects on patent term. 
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Commissioner for Patents 
Box 1450 

Alexandria, VA 22313-1450 
Sir: 



This Amendment is in response to a non-final office action dated April 21, 2004. 
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In the Specification 

Please amend the paragraph beginning at line 1 5 of page 1 2 as follows: 

- The guaranteed minimum motion activity method can now proceed as follows. 
First we assign actual motion activity values, in terms of a continuous descriptor, to 
each level of motion activity. Second, we express the average motion activity of the 
input video as a temporal histogram of the motion activity as described in U.S. 
Patent Application Sn. 09A106,141 10/217.918 11 Activity Descriptor for Video 
Sequence" filed by Divakaran et al. on August 13. 2002. and incorporated herein 
by reference, which is a continuation of now abandoned U.S. Patent Application 
Sn. 09/406,444. filed on September 27, 199 9, incorporated herein by reference . The 
temporal histogram directly indicates what frames of the input video have a level of 
motion activity that is lower than the targeted activity in a quantized fashion so the 
above classification can be performed. Third, we associate the temporal histogram 
with the actual motion values, and apply the guaranteed minimum activity method 
as expressed in the above formulations to determine the relationship between the 
length of the output video and the targeted level of motion activity. -- 

Please amend the paragraph beginning at line 17 of page 8 as follows: 

- Therefore, the system 1 00 samples frames 2 1 1 at a higher rate, and frames 212 
are sampled at a lower rate. In other words, the sampling rate (down-sampling or 
up-sampling) is adaptive to the measured level of motion activity. Low-level 
activities are sped up, and high-level activities are sampled at a normal rate or 
slowed down. In fact, if the level of motion activity is too high to enable normal 
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perception, then the frames 244 212 can be up-sampled. For example, a one second 
sequence of thirty frames can be expanded to a ten second sequence of three- 
hundred frames by showing each frame ten times.-- 
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In the Claims 

1 1 . (currently amended) A method for temporally processing an input video 

2 including a plurality of input frames, each of the input frames having an 

3 associated input frame play time, and the input video having a total input 

4 video play time that is a sum of the input frame play times of all of the input 

5 frames, comprising: 

6 classifying each of the plurality of input frames according to a content 

7 characteristic of each of the input frames; and 

8 allocating an output frame play time to each of the plurality of input 

9 frames that is based on the classified content characteristic of each of the 
10 input frames to generate a plurality of output frames. 

1 2. (original) The method of claim 1 wherein the content characteristic is 

2 based on low-level features of each of the input frames. 

1 3 . (currently amended) The method of claim 1 claim 2 wherein the low-level 

2 features are selected from a group consisting of motion vectors, color, 

3 texture, brightness, contrast, spectral parameters, local and global motion, 

4 activity, trajectory, speed, acceleration, object shape, object size, number of 

5 objects, number of faces, pitch, volume, tempo, and combinations thereof. 

1 4. (original) The method of claim 1 wherein the content characteristic is 

2 based on high-level features of each of the input frames. 
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1 "5. (currently amended) The method of claim 1 claim 4 wherein the high-level 

2 features are selected from a group consisting of genre, dramatic intensity, 

3 humor content, action level, beauty, lyricism, musical intensity, educational 

4 quality, profundity, nudity, linguistic-class, and combinations thereof. 

1 6. (original) The method of claim 1 wherein the allocating of the play time is 

2 dynamically varied while processing the video. 

1 7. (original) The method of claim 1 wherein the allocated output frame play 

2 time of each of the output frames is determined by sampling the input frames. 

1 8. (original) The method of claim 7 wherein the sampling is a down-sampling 

2 of the input frames. 

1 9. (original) The method of claim 7 wherein the sampling is an up-sampling 

2 of the input frames. 

1 10. (original) The method of claim 9 wherein up-sampled output frames are 

2 interpolated from the input frames. 

1 11. (original) The method of claim 7 wherein the sampling is a combination 

2 of down-sampling and up-sampling of the input frames. 

1 12. (original) The method of claim 1 wherein the allocated output frame play 

2 time of each of the output frames is determined by an output frame rate of the 

3 output frame. 
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1 13. (original) The method of claim 12 wherein the output frame rate is 

2 increased for selected input frames. 

1 1 4. (original) The method of claim 1 2 wherein the output frame rate is 

2 decreased for selected input frames. 

1 15. (original) The method of claim 1 further comprising: 

2 measuring the content characteristics of each of the plurality of input 

3 frames to determine the classification. 

1 16. (original) The method of claim 15 further comprising: 

2 computing a statistical moment for the measured content 

3 characteristics to determine the classification. 

1 17. (currently amended) The method of claim 1 wherein the allocation of 

2 play time is based on a constant level of motion activity in the output video 

3 plurality of output frames . 

1 18. (currently amended) The method of claim 1 wherein the allocation of 

2 play time is based on a guaranteed minimum level of activity in the output 

3 video plurality of output frames . 

1 19. (original) The method of claim 1 further comprising: 

2 partitioning the input video into a plurality of segments, and 

3 processing the input video on a per segment basis. 
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1 20. (currently amended) The method of claim 1 wherein still frames are 

2 selected for the output video plurality of output frames when the allocated 

3 output frame play time exceeds a temporal Nyquist limit. 

1 21. (currently amended) The method of claim 1 further comprising: 

2 allocating a total output video play time for an output video: and 

3 allocating the output frame play times so that a sum of the output 

4 frame play times of the plurality of output frames is equal to the total output 

5 video play time of the output video. 

22. (currently amended) The method of claim 1 claim 21 wherein the 
allocated play time of a particular frame can range on a continuum from zero 
time to a length of time of the output video. 

23. (currently amended) The method of claim 1 claim 21 wherein the 
allocation of play time is based on a motion activity in the output video, and a 
measure of motion activity is an average of magnitudes of motion vectors of 
the frames. 

24. (original) The method of claim 23 where the average motion vector 
magnitude ? of theinput video of N frames is expressed as: 

where an average motion vector magnitude of frame i is r,. 
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1 25. (original) The method of claim 24 wherein a relationship between a 

2 length Z output of the output video and a length Linpitt of the input video is 

3 expressed as 



L =— L 

output '-'input 
^target 



5 for a target level of motion activity r target in the output video. 

1 26. (original) The method of claim 25 further comprising: 

2 classifying all of frames j of the input video having the motion activity 

3 equal to or higher than a targeted level of minimum motion activity into a 

4 first set S higher having a length L higher ; 

5 classifying all of frames k of the input video having the motion activity 

6 lower than the targeted level of minimum motion activity into a second S !ower 

7 having a length L, ower ; 

8 summing L higher + L, ower to determine a L input of the input video to 
determine a length of the output video by 



^output 



( '=■ > » 



\ r iarget J 



27. (currently amended) A system for temporally processing an input video 
including a plurality of input frames, each of the input frames having an 
associated input frame play time, and the input video having a total input 
video play time that is a sum of the input frame play times of all of the input 
frames, comprising: 

means for classifying each of the plurality of input frames according to 
a content characteristic of each of the input frames; 
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8 means for allocating a total output video play time for an output video : 

9 and 

1 0 means for allocating an output frame play time to each of the plurality 

11 of input frames that is based on the classified content characteristic of each of 

1 2 the input frames to generate a plurality of output frames so that a sum of the 

1 3 output frame play times of the plurality of output frames is equal to the total 

1 4 output video play time of the output video. 
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Remarks 



Claims 1-27 are pending in the application. Claims 1-27 are rejected. Claims 1,3, 
5, 17-18, 20-23, and 27 are amended. The Specification is amended. No new 
subject matter is added. The rejections are respectfully traversed. 

The Specification is amended to overcome the Examiner's objections. 

Claims 3, 5, 17, 18, 20, and 22-27 are rejected under 35 U.S.C. 112, second 
paragraph as being indefinite. 

Claims 3, 5, 17, 18, 20, 22, 23 and 27 are amended to overcome the rejections. 

Claim 1 is amended to more distinctly claim the invention. 

Claims 1-3, 6-15, 19, 21, and 27 are rejected under 35 U.S.C. 102(e) as being 
anticipated by Abdel-Mottaleb (U.S. 6,424,789 - "Abdel"). 

The invention is frame based. Each frame of an input video has an associated input 
frame play time, and the input video has a total input video play time that is a sum 
of the input frame play times of all of the input frames. Each of the input frames 
of the input video is classified according to a selected content characteristic. An 
output frame play time is allocated to each of the plurality of input frames based on 
the content characteristic of each of the input frames to generate a plurality of 
output frames. 
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Abdel measures differences between frames in a shot to classify the shot. The 
invention looks at only one frame at a time to classify the frame. 

To those of ordinary skill in the art, it is well known that there is a huge difference 
between a 'shot' as described by Abdel, and a 'single frame' as claimed. Abdul 
correctly defines a "shot' (or scene) consisting of a sequence of uninterrupted 
related frames," see Abstract. Applicants assert that a 'sequence' as described by 
Abdul can never be a single frame as claimed. 

That is, a shot includes all frames from when a shutter of a camera opens, until the 
shutter closes. At typical frame rate of 30 or 60 frames per second, it would be 
impossible to perceive a shot as described by Abdel if it were to include only a 
single frame as claimed. Typically, perceivable 'shots' must last from at least 
several seconds to several minutes. This means that 'shots' typically include at least 
hundreds if not thousands of frames. 

Now, Abdul determines a level of activity in a shot, and then performs the selected 
speed change special effect by adding frames or deleting frames to produce a 
modified shot, see Abstract. The invention operates in each frame as it comes by. In 
the most extreme case, and a most liberal interpetation of Abdel the shot must 
include at least a first frame and a second frame, see column 2, 
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45 The video processing device comprises an image processor 
capable of identifying the at least one shot and determining 
a first activity level within at least a first portion of the at 
least one shot, wherein the first activity level indicates a rate 
of change of video content between at least a first frame and 

50 a second frame in the at least a first portion. The image 
processor performs the selected speed change special effect 
by one of adding frames and deleting frames in the at least 
a first portion in response to the first activity level 
determination, thereby producing a modified at least one 

r C shot. 



There is a serious problem with the Abdel system and method. Abdel can never 
operate in real time. This is obvious. Abdel needs to process a shot before a new 
shot can be created. Thus, if the shot is a minute long, the viewer sees nothing until 
at least a minute later, when the result of his processing becomes available. 



In contrast, the invention processes the video and frame at the same time. As each 
input frame is classified, play time is allocated to each corresponding output frame, 
and the output can be viewed, as is, before any more frames need to be processed, 
as in Abdel. 



Abdel is shot based. Abdel classifies shots, see col. 7, lines 33-38, below: 

The shot frames are 
examined to determine the relative degree of movement _ 
between successive frames and/or across a longer series of "° 
frames within the shot. The shot (or a sub-portion of the 
shot) may then be classified according to the level of activity 
in the shot (or sub-portion). 

See also, Figures 3, 4A and 4B. Particularly, see element 320 of Figure 3, element 
415 of Figure 4A, and element 460 of Figure 4B, each describing shot based 
classification. 
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In claims 2-3 , the content characteristic is based on low-level features of each of the 
input frames. The low-level features are selected from a group consisting of motion 
vectors, color, texture, brightness, contrast, spectral parameters, local and global 
motion, activity, trajectory, speed, acceleration, object shape, object size, number of 
objects, number of faces, pitch, volume, tempo, and combinations thereof. The low 
level features for a frame are derived from each input frame for each corresponding 
output frame in the invention. Abdel measures frame differences in shots, which is 
not what is claimed. 



In claim 6, the allocating of the play time is dynamically varied while processing 
the video. In claim 21, a total output video play time is allocated for an output video 
and the output frame play times are allocated so that a sum of the output frame play 
times of the plurality of output frames is equal to the total output video play time of 
the output video. As stated above, play time is frame based in the invention. Abdul 
increases the speed of a shot, see col. 8, 

Once the relative ainoum of movement within the shot is 
35 classified as high to low, the speed of the shot is increased 

in relation to the original shot by saving sub-sarapfcs of 

frames .at a high or k>w rate. If the shot is classified as 

low-level activity, the stib-siirnpk occurs at a comparatively 

low rate (process step 420), The luw sub-sample rate results. 
c , in a relatively smaller number of frames being saved acd a 

relatively larger number of iVanjes being dropped or deleted 

from the original shot. For example, if the activity level is 

low in a (or a sub-ponton of the shot), image processor 120 

may save One out of every three frames and drop the other 
4 - wo frames. 



Above, Abdel classifies the amount of movement in a shot. This is further 
described in col. 9: 
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cmmiltis. Tte r;ite at which frames are dropped during the 
fiist-t'onvurd speed change fe then initially calculated by 
image processor 120 in order to comply svith ihe live minute 
tiirjc constraint, (mage processor 120 may then loop through ±K 
I he video clip one or more lioies lo make the final modified 
video clip play in five minutes or less. linage processor 120 
can apply diflerent weighting factors to high-activity level 
ilud Low-activity level portions of the video Clip in order to 
meet with the specified time duration. - rJ 

There, Abdel must "loop through the video clip one or more times" to meet the 5 
minute constraint. This is a further indication that Abdel cannot be a real time 
process, and that Abdul is a multi-pass operation. In contrast, the invention is a 
single pass operation. 

In claim 7, the allocated output frame play time of each of the output frames is 
determined by sampling the input frames. In claim 8, the sampling is a down- 
sampling of the input frames. In claim 9, the sampling is an up-sampling of the 
input frames. In claim 10, up-sampled output frames are interpolated from the input 
frames. In claim 1 1, the sampling is a combination of down-sampling and up- 
sampling of the input frames. The invention determines the allocated output frame 
play time for each output frame based on the classified content characteristic of 
each of the input frames. Abdel never determines play time. Abdel determines 
activity for an entire sequence of frames in a shot and adds or drops frames based 
on the activity. Abdel can never anticipate what is claimed. 

In claim 12, the allocated output frame play time of each of the output frames is 
determined by an output frame rate of the output frame. In claim 1 3, the output 
frame rate is increased for selected input frames. In claim 14, the output frame rate 
is decreased for selected input frames. The invention determines output frame play 
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time for each output frame. Abdel never changes the frame rate for individual 
frames as claimed. Abdel determines whether the shot is to be played in fast 
forward or slow motion, and adds or drops frames of the shot to achieve a desired 
degree of motion, see col. 6 below: 

-° . Since the 

video transfer rale of the image medium is constant, the 

degree of motion in these shots can be slowed by the 

duplication and addition of selected frames. Conversely, the 

motioH associated with the v.- shorn, can be increased through 
25 the deletion or removal of selected frames *U in accordance 

with the principles of the present invention. 

Claimed is allocating a rate on a per frame basis. Abdel adds or drops frames to 
control a degree of motion in the shots, which has nothing to do with a rate of a 
frame. 

In claim 15, the content characteristics of each of the plurality of input frames is 
measured to determine the classification. Each frame is measured to classify the 
frame. Abdel classifies a shot. These are totally different operations. 

In claim 19, the input video is partitioned into a plurality of segments, and the input 
video is processed on a per segment basis. Therefore, different segments may be 
processed according to different selected content characteristics. The input video is 
still processed on a frame basis, which is never described by Abdel. 

In claim 21, a total output video play time is allocated for an output video and the 
output frame play times are allocated so that a sum of the output frame play times 
of the plurality of output frames is equal to the total output video play time of the 
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output video. As described in col. 9, Abdel must "loop through the video clip one 
or more times" to meet the 5 minute constraint. Abdul is a multi-pass operation. In 
contrast, the invention is a single pass operation. Abdel cannot anticipate the 
invention. 

Claims 4 and 5 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Abdel in view of Mehrotra, et al. (U.S. 6,665,423 - "Mehrotra"). 

Mehrotra generates descriptions of video objects using high-level and low-level 
time varying characteristics of the objects. The invention generates output frames 
having an output frame play time based on the selected content characteristic of 
each of the input frames, which can be high-level features of each of the input 
frames selected from a group consisting of genre, dramatic intensity, humor 
content, action level, beauty, lyricism, musical intensity, educational quality, 
profundity, nudity, linguistic class, and combinations thereof. Mehrotra never 
produces output frames. Mehrotra uses features to generate descriptions of video 
objects. The invention uses features to produce output frames. Mehrotra has 
nothing to do with what is claimed. 

Claim 16 is rejected under 35 U.S.C. 103(a) as being unpatentable over Abdel in 
view of Park, et al. (U.S. 6,597,738 -"Park"). 

Park generates motion descriptors by measuring motion across an entire video or 
respective frames using various first order statistics, see col. 22, below. 
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N r e,\t, tb<! moiion direction descriptor is 10 describe sta- 
tistic characteristics of morion direction with relation to 
moiion data for r&spcclivo images or the whole images in the 
time which is intended by the lime descriptor, and comprised 
of a iTiofion direction avenge, central, moment for the 
average and standard deviation, dominant motion direction, 
accumulated morion histogram and direction data. 



The invention determines higher order statistics for the measured characteristics of 
each frame to determine the classification for each frame to generate an output 
frame. Generating motion descriptors has nothing to do with generating an output 
frame having an output frame play time based on the selected content characteristic 
of an input frame as claimed. Park's motion descriptors can never be used to make 
the invention obvious. 

All rejections have been complied with, and applicant respectfully submits that the 
application is now in condition for allowance. The applicant urges the Examiner to 
contact the applicant's attorney at phone and address indicated below if assistance 
is required to move the present application to allowance. Please charge any 
shortages in fees in connection with this filing to Deposit Account 50-0749 . 



Mitsubishi Electric 
Research Laboratory, Inc. 
201 Broadway 
Cambridge MA, 02139 
(617) 621-7539 



Respectful! 




Andrew J. Curtin 
Reg. No. 48,485 
Attorney for Assignee 
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